SDTP: Accelerating Wide-Area Data Analytics With Simultaneous Data Transfer and Processing

For the efficient analysis of geo-distributed datasets, cloud providers implement data-parallel jobs across sites (e.g., datacenters and edge clusters), which are generally interconnected by wide-area network links. However, current state-of-the-art data analytic methods fail to make full use available computing resources. The main reason is that such must wait for bottleneck complete corresponding transmission computation in each phase. Furthermore, may be impractical bandwidth dynamicity diverse job parallelism. To this end, we propose a Simultaneous Data Transfer Processing (SDTP) mechanism accelerate analytics, with joint consideration dynamics In SDTP, site can execute computation, provided it obtains required input data. As result, loading, map, shuffle, reduce phases at need not completion previous other sites. We further improve SDTP method offering more accurate time estimation generalizing dynamic situations. trace-driven results demonstrate response 19% 72% compared methods.